Crowd-based haptics

ABSTRACT

A system produces haptic effects. The system receives input data associated with an event, identifies an element of the event in the input data, generates the haptic effects based on the element of the event, and produces the haptic effects via a haptic output device. In one embodiment, the haptic effects are generated by haptifying the element of the event. In one embodiment, the haptic effects are designed haptic effects and are adjusted based on the element of the event. In one embodiment, the input data is associated with a crowd that attends the event, and the element of the event is caused by the crowd. In one embodiment, the input data includes haptic data collected by one or more personal devices associated with the crowd. In one embodiment, the input data is indicative of a location of the one or more personal devices associated with the crowd.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of application Ser. No. 14/882,683,filed on Oct. 14, 2015 and issued as U.S. Pat. No. 9,691,238 on Jun. 27,2017, which claims priority of U.S. Provisional Patent Application Ser.No. 62/198,615, filed on Jul. 29, 2015, the disclosures of which arehereby incorporated by reference.

FIELD

One embodiment is directed generally to a haptic system, and inparticular, to a haptic system that provides haptic effects related toan event.

BACKGROUND INFORMATION

“Haptics” relates to a tactile and force feedback technology that takesadvantage of the sense of touch of a user by applying haptic feedbackeffects (i.e., “haptic effects”), such as forces, vibrations, andmotions, to the user. Devices, such as mobile devices, touchscreendevices, and personal computers, can be configured to generate hapticeffects. In general, calls to embedded hardware capable of generatinghaptic effects (such as actuators) can be programmed within an operatingsystem (“OS”) of the device. These calls specify which haptic effect toplay. For example, when a user interacts with the device using, forexample, a button, touchscreen, lever, joystick, wheel, or some othercontrol, the OS of the device can send a play command through controlcircuitry to the embedded hardware. The embedded hardware then producesthe appropriate haptic effect.

Haptics has been leveraged in recent technological advances to enhancethe virtual experience of events such as live sports games, concertshows, fashion shows, comedy gigs, television episodes, etc. To thatend, corresponding haptic effects may be delivered via various means,for example, via traditional media such as radio or television, viaInternet based new media such as news streams and mobile applications,or via event virtual reality platforms such as the Oculus Rift headmounted display (“HMD”) by Oculus Virtual Reality. While suchtechnologies make it possible for a user to “attend” an event remotelyvia a virtual channel, they may not provide a full experience of theambience of a live event as it would have been experienced whenattending the event in person.

SUMMARY

One embodiment is a system that produces one or more haptic effects. Thesystem receives input data associated with an event, identifies anelement of the event in the input data, generates the one or more hapticeffects based on the element of the event, and produces the one or morehaptic effects via a haptic output device. In one embodiment, the hapticeffects are generated by haptifying the element of the event. In oneembodiment, the haptic effects are designed and are adjusted based onthe element of the event. In one embodiment, the input data isassociated with a crowd that attends the event, and the element of theevent is caused by the crowd. In one embodiment, the input data includeshaptic data collected by one or more personal devices associated withthe crowd. In one embodiment, the input data is indicative of a locationof the one or more personal devices associated with the crowd.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a computer server/system in accordance withan embodiment of the present invention.

FIGS. 2-7 illustrate flow diagrams of haptics functionality performed byhaptic systems according to embodiments of the present invention.

FIG. 8 is a flow diagram of the operation of the crowd based hapticsmodule of FIG. 1 when performing haptics functionality in accordancewith embodiments of the present invention.

DETAILED DESCRIPTION

One embodiment provides haptics in content, or more specifically,haptics in live content (or pre-recorded content) to allow a user toexperience the ambience of a live event or a pre-recorded event. Oneembodiment converts an input signal (e.g., an audio signal, a sensorsignal, a signal input by a human operator, etc.) from a live mediabroadcast into a haptic signal that can be used to generate hapticeffects to simulate the ambience of a live or pre-recorded event. In oneembodiment, the haptic signal corresponds to crowd key elements (e.g.,crowd mood, cheers, boos, etc.) or event elements observed by the crowd(e.g., game intensity level, game events, etc.). One embodiment useshaptic data collected by personal devices of the crowd and uses that inproviding remote haptic feedback. Accordingly, by delivering hapticeffects that simulate the experience of being physically present in alive event, embodiments provide an improved virtual experience of anevent.

FIG. 1 illustrates a block diagram of a system 10 in accordance with oneembodiment of the invention. In one embodiment, system 10 is part of amobile device, and system 10 provides haptic conversion functionalityfor the mobile device. In another embodiment, system 10 is part of adevice that is incorporated into an object in contact with a user in anyway (e.g., furniture), and system 10 provides haptic conversionfunctionality for such device. For example, in one embodiment, system 10is part of a wearable device, and system 10 provides a haptic conversionfunctionality for the wearable device. Examples of wearable devicesinclude wrist bands, headbands, eyeglasses, rings, leg bands, arraysintegrated into clothing, or any other type of device that a user maywear on a body or can be held by a user. Some wearable devices can be“haptically enabled,” meaning they include mechanisms to generate hapticeffects. In another embodiment, system 10 is separate from the device(e.g., a mobile device or a wearable device), and remotely provides thehaptic conversion functionality for the device.

Although shown as a single system, the functionality of system 10 can beimplemented as a distributed system. System 10 includes a bus 12 orother communication mechanism for communicating information, and aprocessor 22 coupled to bus 12 for processing information. Processor 22may be any type of general or specific purpose processor. System 10further includes a memory 14 for storing information and instructions tobe executed by processor 22. Memory 14 can be comprised of anycombination of random access memory (“RAM”), read only memory (“ROM”),static storage such as a magnetic or optical disk, or any other type ofcomputer-readable medium.

A computer-readable medium may be any available medium that can beaccessed by processor 22 and may include both a volatile and nonvolatilemedium, a removable and non-removable medium, a communication medium,and a storage medium. A communication medium may include computerreadable instructions, data structures, program modules, or other datain a modulated data signal such as a carrier wave or other transportmechanism, and may include any other form of an information deliverymedium known in the art. A storage medium may include RAM, flash memory,ROM, erasable programmable read-only memory (“EPROM”), electricallyerasable programmable read-only memory (“EEPROM”), registers, hard disk,a removable disk, a compact disk read-only memory (“CD-ROM”), or anyother form of a storage medium known in the art.

In one embodiment, memory 14 stores software modules that providefunctionality when executed by processor 22. The modules include anoperating system 15 that provides operating system functionality forsystem 10, as well as the rest of a mobile device in one embodiment. Themodules further include a crowd based haptics module 16 that provideshaptic functionality, as disclosed in more detail below. In certainembodiments, crowd based haptics module 16 can comprise a plurality ofmodules, where each module provides specific individual functionalityfor providing haptic effects. System 10 will typically include one ormore additional application modules 18 to include additionalfunctionality, such as Integrator™ software by Immersion Corporation.

System 10, in embodiments that transmit and/or receive data from remotesources, further includes a communication device 20, such as a networkinterface card, to provide mobile wireless network communication, suchas infrared, radio, Wi-Fi, or cellular network communication. In otherembodiments, communication device 20 provides a wired networkconnection, such as an Ethernet connection or a modem.

Processor 22 is further coupled via bus 12 to a display 24, such as aLiquid Crystal Display (“LCD”), for displaying a graphicalrepresentation or user interface to a user. The display 24 may be atouch-sensitive input device, such as a touch screen, configured to sendand receive signals from processor 22, and may be a multi-touch touchscreen.

System 10, in one embodiment, further includes an actuator 26. Processor22 may transmit a haptic signal associated with a haptic effect toactuator 26, which in turn outputs haptic effects such as vibrotactilehaptic effects, electrostatic friction haptic effects, or deformationhaptic effects. Actuator 26 includes an actuator drive circuit. Actuator26 may be, for example, an electric motor, an electro-magnetic actuator,a voice coil, a shape memory alloy, an electro-active polymer, asolenoid, an eccentric rotating mass motor (“ERM”), a linear resonantactuator (“LRA”), a piezoelectric actuator, a high bandwidth actuator,or an electroactive polymer (“EAP”) actuator. In alternate embodiments,system 10 can include one or more additional actuators, in addition toactuator 26 (not illustrated in FIG. 1).

Actuator 26 is an example of a haptic output device, where a hapticoutput device is a device configured to output any form of hapticeffects, such as vibrotactile haptic effects, electrostatic frictionhaptic effects, deformation haptic effects, etc., in response to a drivesignal. Accordingly, in alternate embodiments, actuator 26 can bereplaced by some other type of haptic output device (not shown) that maybe a non-mechanical or a non-vibratory device such as a device that useselectrostatic friction (“ESF”) or ultrasonic surface friction (“USF”), adevice that induces acoustic radiation pressure with an ultrasonichaptic transducer, a device that uses a haptic substrate and a flexibleor deformable surface or shape changing device and that may be attachedto a user's body, a device that provides projected haptic output such asa puff of air using an air jet, a laser-based projectile, a sound-basedprojectile, etc.

For example, one embodiment provides a laser-based projectile wherelaser energy ionizes air molecules in a concentrated region mid-air toprovide plasma (a concentrated mixture of positive and negativeparticles). In one embodiment, the laser may be a femtosecond laser thatemits pulses at very fast and very intense paces, and the faster thelaser, the safer for humans to touch. The projectile may appear as ahologram that is haptic and interactive. When the plasma comes incontact with user skin, the user may sense the vibrations of energizedair molecules in the concentrated region. Sensations on the user skinare caused by the waves that are generated when the user interacts withplasma in mid-air. Accordingly, haptic effects may be provided to theuser by subjecting the user to such concentrated region. Alternativelyor additionally, haptic effects may be provided to the user bysubjecting the user to the vibrations generated by directed soundenergy.

Further, in other alternate embodiments, system 10 may not includeactuator 26 or any other haptic output device, and a separate devicefrom system 10 includes an actuator or another haptic output device thatgenerates the haptic effects, and system 10 sends generated hapticsignals to that device through communication device 20.

System 10, in one embodiment, further includes a speaker 28. Processor22 may transmit an audio signal to speaker 28, which in turn outputsaudio effects. Speaker 28 may be, for example, a dynamic loudspeaker, anelectrodynamic loudspeaker, a piezoelectric loudspeaker, amagnetostrictive loudspeaker, an electrostatic loudspeaker, a ribbon andplanar magnetic loudspeaker, a bending wave loudspeaker, a flat panelloudspeaker, a heil air motion transducer, a plasma arc speaker, and adigital loudspeaker. In alternate embodiments, system 10 can include oneor more additional speakers, in addition to speaker 28 (not illustratedin FIG. 1). Further, in other alternate embodiments, system 10 may notinclude speaker 28, and a separate device from system 10 includes aspeaker that outputs the audio effects, and system 10 sends audiosignals to that device through communication device 20.

System 10, in one embodiment, further includes a sensor 30. Sensor 30can be configured to detect a form of energy, or other physicalproperty, such as, but not limited to, sound, movement, acceleration,physiological signals, distance, flow, force/pressure/strain/bend,humidity, linear position, orientation/inclination, radio frequency,rotary position, rotary velocity, manipulation of a switch, temperature,vibration, or visible light intensity. Sensor 30 can further beconfigured to convert the detected energy, or other physical property,into an electrical signal, or any signal that represents virtual sensorinformation. Sensor 30 can be any device, such as, but not limited to,an accelerometer, an electrocardiogram, an electroencephalogram, anelectromyograph, an electrooculogram, an electropalatograph, a galvanicskin response sensor, a capacitive sensor, a hall effect sensor, aninfrared sensor, an ultrasonic sensor, a pressure sensor, a fiber opticsensor, a flexion sensor (or bend sensor), a force-sensitive resistor, aload cell, a LuSense CPS2 155, a miniature pressure transducer, a piezosensor, a strain gage, a hygrometer, a linear position touch sensor, alinear potentiometer (or slider), a linear variable differentialtransformer, a compass, an inclinometer, a magnetic tag (or radiofrequency identification tag), a rotary encoder, a rotary potentiometer,a gyroscope, an on-off switch, a temperature sensor (such as athermometer, thermocouple, resistance temperature detector, thermistor,or temperature-transducing integrated circuit), a microphone, aphotometer, an altimeter, a biological monitor, a camera, or alight-dependent resistor.

In alternate embodiments, system 10 can include one or more additionalsensors, in addition to sensor 30 (not illustrated in FIG. 1). In someof these embodiments, sensor 30 and the one or more additional sensorsmay be part of a sensor array, or some other type of collection ofsensors. Further, in other alternate embodiments, system 10 may notinclude sensor 30, and a separate device from system 10 includes asensor that detects a form of energy, or other physical property, andconverts the detected energy, or other physical property, into anelectrical signal, or other type of signal that represents virtualsensor information. The device can then send the converted signal tosystem 10 through communication device 20.

Generally, with known systems, a user may remotely and virtually attendan event or view a pre-recorded event while receiving correspondinghaptic sensations. For example, motion systems provided by D-BOXTechnologies Inc. provide motion effects in synchronization withspecific visual content to allow for a more realistic experience of thevisual content by a viewer. Such motion systems use the low frequencyaudio content of a media clip, or use human authoring, to create thehaptic effects. Some other known systems such as the “Buttkicker” systemby the Guitammer Company produce haptic effects that enhance thevisual/audio effects felt by a remote user. The Buttkicker system useslow frequency audio content as well as sensors in the stadium to capturethe elements of a gameplay. Some other known systems such as the“Fanmode” application by Fanmode Company allow a remote user to takepart in the stadium interactions of a live sports game by relayingfeedback of the remote user (e.g., waving hands, screaming, etc.) backinto the stadium. With these known systems, while a user can remotelyand virtually attend a live event or a pre-recorded event, he/she atleast partially misses the experience of the ambience of the event thatwould have been experienced by a user that attends the event in person.

In contrast to the known systems, some embodiments of the presentinvention provide haptic feedback that captures elements correspondingto the ambience of an event (e.g., sports events, concerts, shows, etc.)such as dramatic moments and crowd mood (e.g., crowd energy, tension,cheering, etc.). Embodiments re-render this haptic feedback to a remoteuser. One embodiment performs crowd mood inference based on audio,video, and/or sensor signals, or based on data entered by a viewer orattendee of the event, and then provides haptic feedback correspondingto the inferred crowd mood. A further embodiment uses haptic datacollected by personal devices of the crowd and uses that in providingremote haptic feedback. Accordingly, when a user cannot attend an eventin person (e.g., because the event is remote or because the eventhappened in the past), embodiments provide haptic feedback that capturesmore than the audio/visual recorded content of the event so that theuser can see, hear, and also feel what a real attendee would see, hear,and feel.

In one embodiment, data related to the crowd and/or the event iscaptured and then analyzed to detect relevant moments and key elementsthat can be haptically provided to a remote user. Such elements mayinclude, for example, the crowd cheering, booing, chanting at a specificrhythm, shouting at the referee, tapping with their feet, applauding,etc. In one embodiment, these key elements are inferred from capturedaudio signals (e.g., from the main audio feed of the event, from theaudio feeds dedicated to the crowd, from audio or video data recorded bypersonal devices of the crowd, etc.). In one embodiment, audio eventdetection may be performed as described, for example, in U.S. patentapplication Ser. No. 14/078,445, the disclosure of which is incorporatedherein by reference in its entirety.

In one embodiment, once a key element is detected, it is haptified byusing designed effects (e.g., based on a lookup table of designedeffects stored in a database) that can be tuned to match the key elementcharacteristics (e.g., intensity, length, etc.). In additional oralternative embodiments, an identified key element is haptified by usingan audio to haptics conversion algorithm as described, for example, inU.S. patent application Ser. No. 14/078,442, U.S. patent applicationSer. No. 14/078,445, U.S. patent application Ser. No. 14/020,461, and/orU.S. patent application Ser. No. 14/020,502, the disclosure of each isincorporated herein by reference in their entirety.

In one embodiment, other crowd elements may be captured via sensorsplaced at the live event and/or sensors within or attached to personaldevices of the crowd. For example, crowd elements such as peoplestanding up, leaving the event, sitting for a period of time, etc., maybe captured via pressure sensors installed on, under, or around thecrowd seats. Alternatively or additionally, such crowd elements may becaptured and/or identified based on signals captured via personaldevices of the crowd. For example, smartphones of the crowd may be usedto measure signals indicative of the activity of the crowd such asacceleration, deformation, etc. In another example, smartphones of thecrowd may be used to record audio or video signals indicative of theactivity of the crowd.

In an alternative or additional embodiment, various sensors placed atthe live event and/or within or attached to personal devices of thecrowd may be used to sense haptic information related to the event, suchas vibrations caused by a source on a stage, vibrations caused by a racecar that drives by, etc. Accordingly, crowd elements such as the crowdmood may be derived based on signals provided by these haptic sensorsalone, or in combination with signals from other sensors/devices/sourcesdescribed herein. In one embodiment, estimates of the crowd mood may beinferred from each individual signal or from multiple groups of signals,and the crowd mood may then be determined by polling the estimated crowdmoods. For example, in one embodiment, if the polling indicates that 80%of the crowd senses strong haptic effects (e.g., due to a race carpassing by at a certain time), then the crowd mood is determined as“excited.” Alternatively, signals received from multiple sources may bemerged/fused first and then the crowd mood may be inferred from themerged/fused signals.

In one embodiment, such crowd information (e.g., activity of the crowd)as well as data describing player or performer movements (e.g.,accelerations, speed, etc.) and the crowd audio feed (and/or audio datacaptured by personal devices of the crowd) may be used to infer eventcharacteristics such as the mood of the crowd, the game/event dramaticintensity and climax, etc. In one embodiment, such event characteristicsmay be used to adjust the haptic effects created in association with thegameplay or the crowd. For example, in a hockey game, all haptic effects(e.g., haptic effects related to the players, gameplay, crowd, etc.) maybe emphasized (i.e., provided with greater intensity) or reduced (i.e.,provided with less intensity) given the intensity level of the gameitself.

In one embodiment, detection of key elements such as the mood,intensity, and game events is performed using an artificial intelligence(“AI”) based model such as neural networks, support vector machines(“SVMs”), Bayesian networks, etc. The AI based model receives the datafrom the sensors (e.g., pressure sensors, acceleration sensors, audiofeeds, crowd personal devices, etc.), infers elements such as the mostprobable mood or intensity level that resulted in such data, and usesthese elements to tune/adjust the haptic effects related to the game orevent.

In an alternative or additional embodiment, the key elements (e.g.,crowd cheering, booing, etc.) and/or the event characteristics (e.g.,mood, intensity, etc.) are identified by an external observer/curator(i.e., a human operator). The commands of the external observer/curatorand the characteristics of the key element (e.g., magnitude, duration,etc.) are then translated into haptic effects using a pre-configuredlookup table that associates the key elements with correspondingpre-configured haptic effects. Alternatively or additionally, theidentified key elements and event characteristics may be translated intohaptic effects using a corresponding conversion algorithm that covertsaudio or data into haptic effects.

In one embodiment, an event is watched live by the end user, and theaudio and sensory data used to create the haptic effects, the hapticeffects track itself, and/or the operator commands are transmitted inreal time to an end user device. In alternative embodiments, the eventis watched at a later time, and the audio and sensory data used tocreate the haptic effects, the haptic effects track itself, and/or theoperator commands are stored along with the corresponding media contentin order to be displayed/provided to the end user at a later time.

In one embodiment, haptic effects may be generated at different parts ofthe transmission chain of the event. For example, haptic effects may begenerated at the user playback device using sensory/audio data orcommands, or at a remote server that receives the data and the commandsand then transmits the haptic effects to the end user device.Alternatively or additionally, haptic effects may be generated locallyat the sensors and/or within the crowd personal devices. For example,some sensor platforms may have processing capabilities that enable themto locally detect event key elements and/or perform haptic effectsconversion/tuning, such that only the generated/tuned haptic effects aretransmitted from the sensor platforms. In an alternative or additionalembodiment, a dedicated server located at the event venue may receivedata from some or all sensors implemented at the venue and/or from someor all crowd personal devices, and then detect event key elements and/orperform haptic effects conversion/tuning.

In one embodiment, haptic effects related to the crowd or the event maybe provided to a user on any haptic playback media. For example, hapticeffects may be provided at mobile devices (e.g., tablets, smartphones,etc.), wearable devices (e.g., wristband, smart garment, etc.), actuatorequipped furniture, haptically enabled head mounted displays (“HMDs”such as the Oculus Rift), etc. In these embodiments, the audio/videomedia content may be displayed on the same media that provides hapticeffects, or on any other media with playback capabilities (e.g., atelevision).

In one embodiment, an end user can customize the haptic experience tohis/her own taste using a haptic playback device user interface. Forexample, the end user may configure the haptic playback to emphasizesome events, ignore some other events, etc. For example, in oneembodiment, based on user preference to ignore a specific event, thehaptic playback does not create a haptic effect when that event happens.In an alternative or additional embodiment, when a user indicatespreference to ignore an event, the corresponding haptic effect is nottransmitted or played back to the user. Similarly, based on userpreference to emphasize a specific event, haptic effects related to thatevent are not displayed/provided to the user with a higher intensitywhen that event happens.

In one embodiment, an end user may configure the haptic playback toprovide feedback associated with a certain location or point of view(“Pay”) at the event. For example, based on user preferences, only asubset of data from sensors implemented at the venue and/or from crowdpersonal devices may be used to provide the haptic playback. Forexample, when the end user indicates preference for remotelyexperiencing the event as if being present at a certain location at theevent venue, only data from sensors implemented at or around orassociated with that location and/or data from crowd personal devices ator around that location is used to provide haptic feedback to the remoteuser.

In one embodiment, a user may watch a remote or pre-recorded event in ahaptically enabled environment. For example, the user may watch theevent in a haptically enabled room, may use a wearable hapticallyenabled device, may use a Virtual Reality (“VR”) system (e.g., HMDs suchas NextVR or Occulus Rift), etc. When an interesting event occurs duringthe event and corresponds to something that the user cares about (e.g.,a hit during a hockey game), the system identifies that event (e.g.,using sensors or through a curator) and transmits the event or apreprocessed version of the event to the user playback system to providehaptic effects corresponding to the event. Accordingly, the user “feels”the event through the haptic rendering system.

In one embodiment, different “types” of haptic effects may correspond todifferent types of events. For example, in one embodiment, differenthaptic effects can vary and be distinctive to let the user know that thecrowd is excited or relaxed. In an additional or alternative embodiment,one type of haptic effects may be provided with respect to crowd keyelements while a different type of haptic effects may be provided withrespect to performer/player/gameplay events. In one embodiment, a hapticeffect can be varied and thus distinctive from other haptic effects(i.e., different haptic effect types) by varying one or more parameters.In general, high level parameters that define a particular haptic effectinclude magnitude, frequency, and duration. Low level parameters such asstreaming motor commands could also be used to determine a particularhaptic effect. Some variation of these parameters can change the feel ofthe haptic effect, and can further cause the haptic effect to beconsidered “dynamic”. Example types of haptic effects include vibration,jolt, detent, pop, etc., and parameters of each of these haptic effectsmay be changed to result in a different haptic effect. In one example, afirst type of haptic effect can indicate a crowd key element (e.g.,cheering, booing, etc.), while a second type of haptic effect canindicate a gameplay event (e.g., a score, a timeout, etc.). Further,different types of haptic effects can be used to indicate differentinformation, such as the crowd cheering, the crowd booing, etc.

In one example, a user may be interested in a certain music band but maylive too far from a location where the music band is scheduled for alive performance. The user may instead watch a live broadcast of theperformance over the Internet while receiving haptic feedback thatrenders the intensity and energy of the scene. In one embodiment, theuser may watch the performance in the comfort of his/her house, using ahome theater system and a high definition television, while sitting on ahaptic chair that provides haptic effects related to the performance.For example, the chair may shake at the most dramatic moments of theperformance when the crowd chants in unison. In one embodiment, a hapticeffect may be tuned/adjusted by changing a corresponding high levelparameter (e.g., magnitude, frequency, duration, etc.), a correspondinglow level parameter (e.g., streaming motor commands, etc.), or avariation or combination of these parameters to change the feel of thehaptic effect, for example to cause the haptic effect to be considered“dynamic”.

In one embodiment, when inferring the crowd mood, crowd elements such asthe intensity of the event and the excitement of the crowd aredetermined, and corresponding haptic effects are tuned/adjustedaccordingly. In one embodiment, parameters of a haptic effectcorresponding to a key element may be tuned/adjusted according to thecharacteristics of the same key element or a different key element. Forexample, a haptic effect that renders the crowd mood may be tunedaccording to the intensity of the crowd mood elements and/or accordingto the intensity of a game or occurrence of a gameplay event.

In one embodiment, by determining certain events in an audio feed orother event related signal and haptifying only those parts thatcorrespond to those events, an enhanced, targeted, and configurablehaptic feedback is provided to a user. Further, since specific parts ofthe signals are haptified rather than the whole signal, embodiments mayuse haptic conversion algorithms configured for those parts of thesignals and related to corresponding events to provide more accurate andenhanced haptic feedback.

Embodiments are applicable to any live or pre-recorded event.Embodiments are applicable to any audio/video representation of an eventwith a live crowd, where haptic effects are used to render the ambienceof the event as felt by the live crowd.

In one embodiment, event signals such as audio, video, haptic, andsensory signals corresponding to the event or the live crowd may beobtained from user devices of the crowd, e.g., smartphones, wearabledevices, physiological sensors, etc. For example, haptic signals of theevent may be captured by crowd personal devices in the form ofdeformation, acceleration, vibration, audio, etc. In one embodiment,such user devices may communicate event signals via any communicationsmedium such as the Internet, and haptic effects may be generated byobtaining the event signals from the communications medium. For example,in one embodiment, event signals may be captured via a crowd personaldevice (e.g., a microphone, a camera, an accelerometer, etc., which maybe attached to or embedded within a smartphone of a person that attendsthe live event), or may be obtained from contact sensors, pressuresensors, or any physiological sensors associated with one or more usersattending the event.

In one embodiment, the physiological sensors may measure signalsindicative of event key elements such as the crowd mood. For example,the physiological sensors may include a blood pressure sensor or atemperature sensor that provides signals indicative of the level ofexcitement experienced by a person that attends the event. In oneembodiment, other signals may be measured from the crowd body and may beused to obtain physiological signals that are indicative of crowd keyelements. For example, a wearable device may be used to measure theheart rate of a person attending the event, and a rise in the heart ratemay indicate a rise in the level of excitement at the event.Accordingly, a corresponding haptic effect may be provided to a remoteuser.

In one embodiment, user devices of the crowd may be polled to identifyif a subset of the crowd is moving (e.g., if a number of the attendeesare dancing or shaking), and the crowd mood may be inferred accordingly.In one embodiment, movement of the crowd may be measured via a sensor(such as an accelerometer, pedometer, etc.) configured within a wearableor a handheld user device. In one example, the crowd mood may beinferred as being excited if the number of attendees that are movingenergetically is more than a threshold. In one embodiment, one or moreattendees may allow crowd information to be collected via their userdevices and/or wearables, and crowd information may be pulled only fromthose devices that are authorized by their users to share information.

In one embodiment, crowd key elements may be obtained/inferred via eventinformation that is uploaded and/or shared by attendees viacommunication networks such as Twitter, Facebook, etc. In oneembodiment, uploaded/shared event information related to a certain eventis identified by observing event identifiers associated with the eventsuch as a hashtag, location information, event name, event date/time,etc. In one embodiment, corresponding haptic effects for some event keyelements are determined based off of one or more recordings associatedwith an event, and then such effects are re-used when rendering hapticeffects for similar event key elements for the same event or for otherevents.

In one embodiment, when a user creates and shares a video of an event,the user is given the option to provide recommendations/suggestions forassociating haptic effects with the video or with one or more portionsof the video. In one embodiment, the user may select a haptic effectfrom a set of pre-recorded effects and associate that haptic effect withthe uploaded/shared video or with one or more portions of theuploaded/shared video. In one embodiment, the user may further recordhaptic event information with the same device used for creating thevideo or with a different device, and share the haptic information as ahaptic track associated with the video or as haptic information that maybe used to create a haptic track associated with the video.

One embodiment uses a number of uploaded/shared videos of an event togenerate haptic effects related to event key elements of the eventcaptured by those videos. In one embodiment, such haptic effect isassociated with future uploaded/shared videos that capture same orsimilar event key elements in the same or a different event. Oneembodiment uses captured haptic information of multiple crowd personaldevices to generate a haptic track of the event and/or to generatehaptic effects related to event key elements of the event captured bythe haptic information.

In one embodiment, event key elements are inferred from eventinformation that is uploaded/shared by the attendees and/or other remoteviewers. In one embodiment, event key elements may be inferred by usingdata fusion functionality and/or correlating various sources ofinformation such as audio feeds, video feeds, attendee/player/gameplaysensors configured at the event, uploaded videos of the event (e.g.,videos that can be associated with the event based on a correspondinghashtag), comments made on uploaded videos that are indicative of anevent key element (e.g., “the crowd is cheering!”), the timestamp of acertain comment made on an uploaded video, tweets related to an eventand made at certain timestamps, information sourced by crowd personaldevices, etc. In one embodiment, the aforementioned information iscorrelated and used to infer various events and/or create correspondinghaptic effects.

In one embodiment, a number of uploaded/shared videos or haptic tracksof an event are processed to obtain a model of a certain event or acertain event key element. In one embodiment, such model is updateddynamically upon availability of new uploaded/shared videos or haptictracks and is applied to future videos or haptic tracks to identifycorresponding events or event key elements and potentially tune/adjustthe model.

One embodiment provides a haptic playback system for a broadcasted liveevent, and haptic effects are provided to remote users that receive thebroadcast of the live event. In one embodiment, the haptic playbacksystem includes an end user device that receives separates signals forvideo data of the live event and haptic data corresponding to the videodata, or receives a composite signal including the video data and thehaptic data. The haptic data may include a direct haptic effect streamor a set of commands indicating which haptic effects must be performed.

FIG. 2 illustrates a flow diagram of haptics functionality performed bya haptic system 200 according to an embodiment. At 202 one or more audiofeeds of an event are received and at 204 acoustic (i.e., audio) eventsdetection is performed on the received audio data. Alternatively oradditionally, audio data may be received at 203 from one or more crowddevices (e.g., personal devices of the crowd that attends the event).Based on the audio events detection, at 206 key elements of the eventare identified. In one embodiment, the key elements may correspond tocrowd key elements such as cheers, boos, etc. At 208 audio to hapticsconversion is performed only for the detected crowd key elements withinthe audio signal of the event, and at 210 corresponding crowd hapticeffects are provided to an end user.

FIG. 3 illustrates a flow diagram of haptics functionality performed bya haptic system 300 according to an embodiment. Haptic system 300performs the same functionalities at 202, 203, 204, 206, and 210 as inhaptic system 200 of FIG. 2. However, in haptic system 300, once thecrowd key elements are identified at 206, at 302 a pre-set effect istuned and injected for each identified crowd key element. The pre-seteffect may correspond to characteristics of a haptic effect that will beprovided to a user in response to identifying a certain crowd keyelement. For example, for each identified crowd key element such ascheers, boos, etc., a pre-set effect such as intensity, duration, etc.,may be determined, and a haptic effect with such pre-set effects isprovided to an end user at 210.

FIG. 4 illustrates a flow diagram of haptics functionality performed bya haptic system 400 according to an embodiment. At 402, 404, 406, 424,and 426, event information is received via crowd sensors and/orplayers/gameplay sensors and/or one or more audio feeds and/oruploaded/shared information and/or crowd devices, respectively. In oneembodiment, crowd sensors may include pressure sensors placed on, under,or around crowd seats. In one embodiment, uploaded/shared informationincludes, for example, information shared by attendees of the event viatheir personal communication devices, information shared by viewers ofthe event on social networks, videos of the event uploaded on theInternet, tweets related to the event, comments on videos or tweetsrelated to the event, etc. At 408 audio analysis is performed on theaudio feed signals and/or audio data received from crowd devices. In oneembodiment, audio analysis includes acoustic events detection. In analternative embodiment, audio analysis includes detecting one or moreaudio attributes such as the audio feed average volume/intensity, aspecific frequency range content, etc.

At 410 the event signals from 402, 404, 408, 424, and 426 are fed to anAI decision support model to identify various key elements. At 412, 414,and 416 event key elements such as crowd mood and/or game intensitylevel and/or game events are identified, respectively. At 418 theidentified key elements are used to tune a haptic track obtained from420. For example, in one embodiment, the identified key elements may beused to tune the intensity and duration of specific haptic effects inthe haptic track. In one embodiment, the haptic track may be authored ormay be obtained by conversion from audio/video/sensor/haptic informationfrom the event. Finally, at 422 the tuned haptic track is provided to anend user.

FIG. 5 illustrates a flow diagram of haptics functionality performed bya haptic system 500 according to an embodiment. Haptic system 500performs the same functionalities at 206, 208, and 210 as in hapticsystem 200 of FIG. 2. However, in haptic system 500, at 502 eventinformation is curated by a human operator and provided to 206. In oneembodiment, the human operator may be onsite at the event or may bereceiving audio/video feeds from the event. In one embodiment, the humanoperator provides feedback that flags events corresponding to crowd keyelements such as cheers, boos, etc.

FIG. 6 illustrates a flow diagram of haptics functionality performed bya haptic system 600 according to an embodiment. Haptic system 600performs the same functionalities at 502, 206, and 210 as in hapticsystem 500 of FIG. 5. However, in haptic system 600, once the crowd keyelements are identified at 206, at 302 a pre-set effect is tuned andinjected for each identified crowd key element as described herein withreference to FIG. 3, and a haptic effect with such pre-set effects isprovided to an end user at 210.

FIG. 7 illustrates a flow diagram of haptics functionality performed bya haptic system 700 according to an embodiment. Haptic system 700performs the same functionalities at 412, 414, 416, 418, 420, and 422 asin haptic system 400 of FIG. 4. However, in haptic system 700, at 502event information is curated by a human operator as described hereinwith reference to FIG. 5. Such event information is then used toidentify event key elements in 412, 414, and 416.

One embodiment uses crowd personal devices (e.g., smartphones, wearabledevices, etc.) to record a haptic track (e.g., vibrations) forlater/remote playback along with a video track of the event. In oneembodiment, the crowd is directed to associate a video/audio track withthe haptic track recorded by a respective device. One embodiment mayimplement an array of haptic sensors at one or more of such crowdpersonal devices. One embodiment provides functionality for collectingand merging the vibration data generated by a large number of suchdevices present in the crowd (e.g., at a music concert, at a car race,etc.). The multiple recordings may be used to improve the quality of thehaptic track and/or provide localized haptic data for playback in VR.Accordingly, if the sensors available in a single crowd user devicecannot provide an accurate reproduction of a haptic experience, hapticdata recorded by multiple crowd personal devices is used/merged toprovide a better haptic track. This advantage is even more significantwhen it is expensive and difficult to deploy more accurate recordingequipment. Further, multiple haptic track recordings may be necessarywhen haptic experience at different locations is desired to bereproduced, for example, in contexts such as VR.

In one embodiment, the sensors available on smartphone and wearabledevices (e.g., tablets, smart watches, etc.) are used to generate ahaptic track for audio-visual content. The sensors may include, forexample, accelerometers, microphones, dedicated haptic recording devices(e.g., laser vibrometers, interferometry vibrometers), etc.Alternatively or additionally, haptic information may be derived basedon video motion estimation (e.g., based on the change between successiveframes captured by a smartphone or a wearable device). One embodimentfurther uses data captured by other devices distributed at the eventsuch as professional devices that use the same sensor technologies as inconsumer devices (e.g., accelerometers, microphones, laser vibrometers,etc.) but are more expensive and have better performance (e.g., betterprecision, resolution, signal-to-noise ratio, sampling rate,directionality, etc.).

One embodiment provides functionality for collecting haptic data frommultiple user devices in a crowd (e.g., at a music concert, at a carrace, etc.). The multiple recordings can then be merged, for example, inpost-processing or in real-time. In one embodiment, the merged recordingmay be used to generate a haptic track with a higher quality than itsconstituent haptic tracks. For example, the multiple recordings may beaveraged to reduce noise, or a subset of the multiple recordings may beselected as the best recordings and then averaged or merged.Alternatively, a data fusion algorithm may be used to merge the multiplerecordings as will be described herein with reference to variousembodiments.

In one alternative or additional embodiment, the merged recording may beused to generate one or more spatialized haptic tracks for context suchas VR. The haptic tracks may then be associated with an audio-visualrecording of the experience. In this embodiment, a haptic track may begenerated by interpolating between a number of recordings related to thevirtual position being simulated in VR. For example, the haptic trackmay be generated based on the recordings that are closest to the virtualposition being simulated in VR, based on a number of recordings aroundthe virtual position being simulated in VR, etc. In one embodiment, amodel of propagation may be used to predict the haptic recording at aspecific spatial location. For example, the model of propagation may beused to reconstruct the source of haptic feedback based on one or morehaptic recordings, and then predict the haptic effect as it will beexperienced at the specific spatial location.

One embodiment collects and merges recordings of several crowd personaldevices by using a cloud service. For example, in one embodiment, eachcrowd personal device records a haptic track and communicates therecording to a server within a cloud service. The server then mergesand/or averages the received recordings and provides that to a playbackdevice of a remote user.

In one embodiment, various haptic tracks collected/recorded by variouscrowd personal devices are synchronized before being merged/averaged.For example, in one embodiment, each crowd personal device timestampsits recordings and the timestamps are used to synchronize the recordingsof various crowd personal devices. In one embodiment, data collected bycrowd personal devices is transferred to a server or central device foranalysis, along with location information and metadata about thetransferred recordings. The transferred data may include the timestampsto facilitate synchronization with other recordings so that embodimentsmay determine the timing relationship between various measurementsrecorded at various locations and/or by variousdevices/wearables/sensors.

One embodiment detects missing information (e.g., gaps) in therecordings received from crowd personal devices and directs members ofthe crowd that are participating in the recording towards spots withweak coverage/recordings or no coverage/recordings. One example ofdetecting missing information in the recordings received from the crowdand directing members of the crowd accordingly is disclosed inSchofield, et al., “Bootlegger: Turning Fans into Film Crew,” CHI 2015Proceedings of the 33rd Annual ACM Conference on Human Factors inComputing Systems, Pages 767-776 (“Schofield”). The directing of thecrowd may be based on a desired location of the crowd (e.g., move to theleft), a desired subject (e.g., shoot the singer or drummer), etc.

In one embodiment, the directing of the crowd is performed by providingfeedback to the crowd (e.g., visual feedback, haptic feedback, etc.).One embodiment collects different types of recordings from the crowd,such as audio recordings, video recordings, haptic recordings, etc. Forexample, a crowd personal device may provide both a video recording andan associated haptic recording, and the haptic recording may be used toprovide remote haptic feedback along with a playback of the videorecording. In one embodiment, the directing of the crowd may beaccording to the preferences of a remote viewer of an event. Forexample, one or more live attendees of an event may serve as a proxy fora remote viewer of a broadcast of the event. For example, the remoteviewer may determine which live haptic recording is to be forwarded forplayback, based on a location of an attendee, a subject captured by therecording of an attendee, etc. In one embodiment, a remote viewer mayalternatively or additionally select a player/performer to receivefeedback from. For example, a remote viewer may select to receive hapticand/or other feedback captured by a sensor/device attached to a playerof a soccer game, and the viewer may change the selection at a latertime to get feedback from a different player and/or from a live attendeeof the game.

In one embodiment, for example, a person in a crowd that is attending aconcert may use an application on a personal device (e.g., a smartphone)to record a haptic track of the performance. The user may thenre-experience the event at a later time by using Oculus Rift. Forexample, the person may virtually move through the crowd and feel thehaptic experience change as if being present at the event.

In one embodiment, for example, a user may watch a car race event ontheir tablet, and as the race cars drive by, feel the vibrations as ifbeing within the live crowd. Such vibrations may have been recorded bymultiple crowd user devices (e.g., smartphones) at the race track.Accordingly, in broadcasting the event to be watched by the person, thebest vibration recordings may be selected and provided based on thecamera used at a certain time. For example, when the cameras are focusedon a certain location in the race track, the vibration recordings fromcrowd personal devices that are close to that certain location and/orobserving that certain location may be used to provide haptic feedbackto the person that is watching the broadcast of the event.

In one embodiment, the crowd personal devices determine their positionin the three dimensional space (i.e., provide spatial feedback). Thismay be performed via outdoor positioning systems such as the globalpositioning system (“GPS”) or by implementing indoor positioningfunctionality such as detecting proximity Bluetooth beacons distributedin the space. One alternative or additional embodiment may estimate therelative position of different devices by having them communicate withone another (e.g., by WiFi Direct or Bluetooth) and estimating thesignal strength. One alternative or additional embodiment may estimatethe position of the devices based on other sensor signals such as imagescaptured by a camera or the intensity of the sound captured by thedevice. For example, the location of a crowd personal device may bedetermined based on light intensity at that location in images capturedby a camera. In another example, the location of a crowd personal devicemay be determined based on pattern recognition in images captured by acamera. In another example, the location of a crowd personal device maybe determined based on sound intensity as captured by the deviceindicating the source of the sound and/or how far the device is from thestage.

In one embodiment, after various event data are collected (e.g., fromdifferent sensors at different locations at the event, from crowdpersonal devices, etc.), such data is fused into a coherent data set forlater and/or remote playback. One embodiment fuses the collected datainto a single haptic track that can be played back along with acorresponding audio-visual content, thereby providing a single POVfeedback. An alternative or additional embodiment uses the collecteddata to produce a map of vibrations based on various locations at theevent, thereby providing multiple POV feedback.

In one embodiment, in order to provide a single POV feedback, thecollected data is combined such that an optimal resulting haptic trackis obtained. The combining may be performed using any sensor fusionalgorithm known in the art that perform data fusion based on, e.g.,noise or variance of the signal recorded by each sensor/device,characteristics of each sensor (e.g., signal-to-noise ratio, recordingquality rating, etc.), amplitude of the recorded signal, distance fromthe source of the recorded vibrations (e.g., the stage or speakers at aconcert), etc. For example, in one embodiment, the sensor fusionfunctionality may obtain a weighted average of sensor signals, where theweights are based on the quality rating of each sensor.

One alternative or additional embodiment provides multiple POV feedbackthat maps the vibrations felt at various locations. This embodiment isapplicable, for example, in a VR context where a user can look aroundand/or move in the space. It is also applicable in non-VR contexts wherecamera angles can be selected by the user that is receiving hapticfeedback, and the haptic feedback is provided in association with theselected camera (e.g., different haptic tracks are provided fordifferent video tracks). In this embodiment, the sensor fusionfunctionality may be as in the single POV scenario, but only applied tocollected data that corresponds to certain locations at the event. Forexample, in order to provide haptic feedback corresponding to a specificlocation (e.g., a point in the space), one embodiment determines thebest fusion of recordings from sensors/devices that are near thatlocation (e.g., sensors/devices that are in a certain vicinity of thatlocation). Accordingly, this embodiment provides a set of haptic tracksfor some positions in space, from which a haptic track may be derivedfor any point in space (e.g., based on interpolation).

One alternative embodiment, however, may provide multiple POV feedbackby using the collected data to determine a single fused haptic track anda map of the intensity of vibrations at various spatial locations. Thisembodiment produces haptic feedback for a certain spatial location basedon the fused haptic track and the intensity of vibrations at thatcertain spatial location (e.g., by scaling the fused haptic track basedon the measured or predicted intensity of vibrations at that certainspatial location).

One embodiment implements multiple actuators in providing hapticfeedback. For example, one or more haptic feedbacks derived via thefunctionality described herein with respect to multiple POV feedback maybe assigned to respective actuators of a device that has multipleactuators. This embodiment is applicable, for example, in providingplayback on a tablet with actuators on the left and right side,providing playback through wearables on the left and right wrists of auser, etc. In one embodiment, the functionality described herein withreference to multiple POV feedback may be implemented to derive onehaptic track per actuator. For example, based on data collected at theevent by various sensors/devices at various locations, if strongervibrations are sensed at the left side of a location of interest, theleft-hand actuator of a tablet may provide a stronger playback vibrationcompared to its right-hand actuator.

In one embodiment, the haptic feedback is provided along withaudio-visual and/or VR content. The audio-visual and/or VR content maybe provided, for example, on a television, smartphone, tablet, VRheadset, etc. The accompanying haptic feedback may be provided, forexample, on a smartphone, a tablet, a wearable (e.g., smart watches),etc. In one embodiment, the haptic feedback may be according to any typeof haptic stimuli described herein (e.g., vibrations, poking, squeezing,deformation, etc.) and is generated based on a mapping between therecorded vibrations and the respective haptic stimuli. For example, inone embodiment, the intensity of the recorded vibrations may be mappedto the intensity of deformations on a playback device. In oneembodiment, the playback device receives the audio-video content alongwith the haptic data and plays them back in synchrony.

In one embodiment that provides the functionality described herein withreference to multiple POV feedback, a playback client may identify ahaptic track for playback based on position/orientation of a remote userin a VR environment. For example, the playback client may identify thehaptic track by linearly interpolating between the two nearest hapticdata points provided by data fusion functionality.

One embodiment may implement an application on one or more crowd userdevices that collects data from such devices. The application may alsooptionally provide guidance to improve the quality of the recordingsperformed via crowd user devices. For example, by performing at leastpart of sensor fusion functionality based on data that is alreadycollected from sensors/devices at an event, one embodiment determines anestimate of the current quality of the recordings. If the estimateindicates that there is a gap in the recordings corresponding toinadequate or missing information related to a certain spatial locationat the event, the application directs one or more users in the crowd tomove towards the corresponding spatial location in order to improve thecoverage of that area.

In one embodiment, user smartphones and/or wearables in a crowd at aconcert are used to record vibrations. The recordings are then collectedby a server and the collected data is processed to generate ahigh-quality haptic track that optimally combines all the recordings.Alternatively or additionally, the collected data is processed togenerate a spatialized haptic track that reproduces the hapticexperience at different locations in the crowd.

One embodiment provides data fusion functionality for fusing datacollected from various sensors/devices at an event. In one embodiment,the sources of data may be redundant (e.g., two or more sources providethe same data), cooperative (e.g., fusion of data from two or moresources provides data that is more accurate than each source), orcomplementary (e.g., data from different sources corresponds todifferent parts of the environment), depending on the nature anddistribution of the used sensors. The data fusion functionality may beaccording to, for example, Bayesian inference, maximum likelihood, leastsquares, Kalman filters, particle filters, ensemble methods, etc.Bayesian inference is a statistical inference in which Bayes' theorem isused to update the probability for a hypothesis as evidence is acquired.Bayesian inference may be Laplacian based, Gaussian based, etc. Maximumlikelihood includes finding the value of one or more parameters for agiven statistic which maximizes its likelihood distribution. The methodof least squares is the process of finding the solution that minimizesthe sum of the squares of the errors. Least squares may be performed asunscented, weighted, etc. Kalman filters use a series of measurementsobserved over time and including statistical noise and otherinaccuracies, and produce estimates of unknown variables that are moreprecise than those that would be obtained based on a single measurementalone. Kalman filters may be extended, unscented, etc. Particle filtersmethod is a sequential Monte Carlo methods based on point mass (or“particle”) representations of probability densities. Ensemble methodsuse multiple learning algorithms to obtain better predictive performancethat could be obtained from any of the constituent learning algorithms.

In one embodiment, the choice of the data fusion functionality may bebased on data characteristics such as the number of recordings, therelative location of the recordings, the amount of noise in therecordings, the type of noise in the recordings (e.g., Gaussian, random,etc.), etc. For example, when different observations from differentsources/recordings/devices with possibly various errors are used topredict the same variable (e.g., the vibration at a certain location),data from various sources may be input into a Kalman filter which, givena model of vibration propagation, can determine the best prediction ofthe valuable at a location that has no corresponding recording. In oneembodiment, the observation that is input into a data fusion algorithmwith reference to a specific recording may be the root mean square(“RMS”) value of the recorded signal within a certain time period (e.g.,a 5 ms window), the maximum value of the signal within a certain timeperiod, etc. In one embodiment, each value delivered by one of thesensors/devices concerning the acceleration/vibration (e.g., a measuredacceleration value) is considered as an observation of a correspondingstate variable.

The data fusion functionality described herein may be implemented forany playback configuration such as providing a single POV feedback,providing multiple POV feedback, assigning one or more different haptictracks to respective actuators of a device with multiple actuators, etc.In one embodiment, in order to provide multiple POV feedback and/orassign different haptic tracks to different actuators, for eachsensor/device located in a specific area at an event, the correspondingreadings are considered as observations of the state variablecorresponding to that location. Depending on the type of the data fusionfunctionality used, one embodiment also applies a modeling of theevolution of the state variable over time. An evolution model may be anequation system that links the state variable values at time “t” totheir values at time “t−1,” and may depend on the data being used andthe variables that are being estimated. One alternative or additionalembodiment uses a data fusion functionality as described herein toestimate haptic information (e.g., the value of acceleration/vibration)in areas where no sensed/recorded information is available but relatedsensed/recorded information is gathered from sensors/devices availableat a surrounding area.

In one embodiment that implements a Kalman filter for data fusion, inorder to fuse the collected data into a single haptic track to provide asingle POV feedback, each sensor/smartphone/device around a point Pprovides an observation of the state variable that is to be estimated,which is the vibration level at point P. Assuming that the process noiseand the observation noise are both white Gaussian noises with knowncovariance matrices and the state variable follows a Gaussiandistribution, the state variable evolves in time by the addition of thewhite noise. In this embodiment, the observation model also integratesthe vibration propagation model that determines the vibration at acertain distance from the source of the vibration.

FIG. 8 is a flow diagram of crowd based haptics module 16 of FIG. 1 whenproducing one or more haptic effects in accordance with embodiments ofthe present invention. In one embodiment, the functionality of the flowdiagram of FIG. 8 is implemented by software stored in memory or othercomputer readable or tangible medium, and executed by a processor. Inother embodiments, the functionality may be performed by hardware (e.g.,through the use of an application specific integrated circuit (“ASIC”),a programmable gate array (“PGA”), a field programmable gate array(“FPGA”), etc.), or any combination of hardware and software.

At 802 crowd based haptics module 16 receives input data associated withan event. In one embodiment, the input data is originated from one ormore of an audio feed, a video feed, a sensor, a human operator, awebsite, or a user device of an attendee of the event. In oneembodiment, the input data is associated with a crowd that attends theevent. In one embodiment, data associated with the crowd includes datasourced/generated by the crowd (e.g., cheers, boos, chants, etc.) and/ordata that captures the event as experienced by the crowd (gameintensity, mood, player performance, etc.).

At 804 crowd based haptics module 16 identifies an element of the eventin the input data. In one embodiment, the input data is originated fromtwo or more sources of data, and the element of the event is identifiedby correlating information in the two or more sources of data. In oneembodiment, the element of the event is caused by the crowd. In oneembodiment, the element of the event may correspond to crowd mood, suchas the crowd being excited, agitated, dancing, etc. In one embodiment,the element of the event corresponds to a distinct crowd noise, such ascheering, booing, gasping, etc.

At 806 crowd based haptics module 16 generates one or more hapticeffects based on the element of the event. In one embodiment, thegenerating of the one or more haptic effects may include adjusting theone or more haptic effects. In one embodiment, the one or more hapticeffects are generated by haptifying the element of the event. In analternative embodiment, the one or more haptic effects are designedhaptic effects that are adjusted based on the element of the event. Inone embodiment, the adjusting includes tuning a parameter (e.g., anintensity, a duration, etc.) of the one or more haptic effects based onthe element of the event. In one embodiment, the one or more hapticeffects are generated based on a different element of the event or basedon a different input data associated with the event, where the differentelement of the event is obtained based on the input data or based on thedifferent input data.

At 808 the one or more haptic effects are produced via a haptic outputdevice. In one embodiment, the input data captures the experience of anattendee of the event located at a specific location and/or viewing theevent from a specific POV/angle. Accordingly, such haptic effects helpthe user experience the event as if being located at such locationand/or as if viewing the event from such specific POV/angle. In thisembodiment, the haptic effects may be modified and/or adjusted based ona location/POV/angle selected by the user among variouslocations/POVs/angles at the event. In one embodiment, input data forproviding haptic effects corresponding to each location/POV/angle iscollected from the crowd present at that location/POV/angle and/or fromsensors capturing event information at that location/POV/angle.

In one embodiment, the input data includes haptic data collected by oneor more personal devices associated with the crowd that attends theevent. In one embodiment, the input data further includes video or audiodata collected by one or more personal devices associated with the crowdthat attends the event. In one embodiment, the input data is indicativeof a location of the one or more personal devices associated with thecrowd that attends the event. In one embodiment, upon receiving theinput data, crowd based haptics module 16 determines that the hapticdata is missing haptic information associated with a location at theevent and directs one or more of the crowd to move toward the location.In one embodiment, upon receiving the input data, crowd based hapticsmodule 16 determines that the haptic data is missing haptic informationassociated with a performer at the event and directs one or more of thecrowd to capture the haptic information about the performer.

In one embodiment, crowd based haptics module 16 generates a haptictrack of the event based on the haptic data collected by the one or morepersonal devices associated with the crowd that attends the event. Inone embodiment, crowd based haptics module 16 generates one or morehaptic tracks associated with respective different locations at theevent based on the haptic data that is collected by the one or morepersonal devices associated with the crowd that attends the event.

In one embodiment, crowd based haptics module 16 determines that aremote user indicates preference for receiving haptic feedbackassociated with a certain location at the event. In one embodiment,crowd based haptics module 16 selects, based on the preference, a haptictrack within the one or more haptic tracks, and provides haptic feedbackto the remote user based on the haptic track.

As disclosed, embodiments allow for a live event ambience to beexperienced by a remote user. One embodiment captures event data (e.g.,audio, video, sensory data, uploaded/shared event information, etc.) andanalyzes the event data (either automatically or by a human operator) toidentify event key elements to be haptified. One alternative oradditional embodiment uses the captured data to obtain parameters usedto tune haptic effects related to event key elements. Embodimentsconvert relevant captured data into haptic effects (either by automaticconversion or based on designed effects). One embodiment stores the dataused to generate or tune the haptic effects, and/or the haptic effectstrack itself, so that the stored data and/or the haptic track can beprovided to a user playback device at a later time. An alternativeembodiment transmits the data used to generate or tune the hapticeffects, and/or the haptic effects track itself, to a user playbackdevice. The haptic effects are then provided to the end user along withthe media content of a live or registered event. Accordingly,embodiments give the end user the possibility of better virtualimmersion into the ambience of a remote or pre-recorded event.

Several embodiments are specifically illustrated and/or describedherein. However, it will be appreciated that modifications andvariations of the disclosed embodiments are covered by the aboveteachings and within the purview of the appended claims withoutdeparting from the spirit and intended scope of the invention.

What is claimed is:
 1. A method of producing one or more haptic effects,comprising: receiving, from a mobile phone at an event, input data thatincludes audio content captured by the mobile phone and associated withthe event; identifying an element of the event based on the audiocontent in the input data; generating the one or more haptic effectsbased on the element of the event; and playing back the one or morehaptic effects via a haptic output device, wherein the playing back ofthe one or more haptic effects is performed in synchronization withplaying back the audio content of the event.
 2. The method of claim 1,wherein the one or more haptic effects are generated by haptifying theelement of the event.
 3. The method of claim 1, wherein the generatingcomprises tuning a parameter of the one or more haptic effects based onthe element of the event.
 4. The method of claim 1, wherein the one ormore haptic effects are designed haptic effects that are adjusted basedon the element of the event.
 5. The method of claim 1, wherein the inputdata is originated from two or more sources of data, wherein the elementof the event is identified by correlating information in the two or moresources of data.
 6. The method of claim 1, wherein the mobile phone isone of a plurality of mobile phones associated with a crowd at theevent.
 7. The method of claim 1, wherein the element of the event iscaused by a crowd that attends the event.
 8. The method of claim 1,wherein the input data is indicative of a location of the mobile phone,wherein the mobile phone is one of a plurality of mobile phonesassociated with a crowd at the event.
 9. The method of claim 1, whereinthe input data further comprises haptic data collected by the mobilephone, wherein the mobile phone is one of a plurality of mobile phonesassociated with a crowd at the event.
 10. The method of claim 9, furthercomprising: determining that the haptic data is missing hapticinformation associated with a location at the event; and directing oneor more of the crowd to move toward the location.
 11. The method ofclaim 9, further comprising: determining that the haptic data is missinghaptic information associated with a performer at the event; anddirecting one or more of the crowd to capture the haptic informationabout the performer.
 12. The method of claim 9, further comprising:generating a haptic track of the event based on the haptic datacollected by the mobile phone.
 13. The method of claim 9, furthercomprising: generating one or more haptic tracks associated withrespective different locations at the event based on the haptic datacollected by different mobile phones at the respective differentlocations.
 14. The method of claim 13, further comprising: determiningthat a remote user indicates a preference for receiving haptic feedbackassociated with a certain location at the event.
 15. The method of claim14, further comprising: selecting, based on the preference, a haptictrack within the one or more haptic tracks; and providing hapticfeedback to the remote user based on the haptic track.
 16. Anon-transitory computer readable medium having instructions storedthereon that, when executed by a processor, cause the processor toproduce one or more haptic effects, the processor: receiving, from amobile phone at an event, input data that includes audio contentcaptured by the mobile phone and associated with the event; identifyingan element of the event based on the audio content in the input data;generating the one or more haptic effects based on the element of theevent; and playing back the one or more haptic effects via a hapticoutput device, wherein the playing back of the one or more hapticeffects is performed in synchronization with playing back the audiocontent of the event.
 17. The non-transitory computer readable medium ofclaim 16, wherein the one or more haptic effects are generated byhaptifying the element of the event.
 18. A system for producing one ormore haptic effects, the system comprising a processing circuitconfigured: to receive, from a mobile phone at an event, input data thatincludes audio content captured by the mobile phone and associated withthe event; to identify an element of the event based on the audiocontent in the input data; to generate the one or more haptic effectsbased on the element of the event; and to play back the one or morehaptic effects, wherein playing back the one or more haptic effects isperformed in synchronization with playing back the audio content of theevent.
 19. The method of claim 1, wherein the input data furtherincludes video content captured by the mobile phone and associated withthe event, wherein the element is identified based on the audio contentand the video content, and wherein the one or more haptic effects areplayed in synchronization with the audio content and the video content.20. The method of claim 1, wherein identifying the element of the eventcomprises identifying a crowd characteristic of a crowd of the event,and wherein generating the one or more haptic effects based on theelement of the event comprises determining an amplitude, frequency, orduration of the one or more haptic effects based on the crowdcharacteristic of the event.